Auto calibration and personalization of eye tracking system using larger field of view imager with higher resolution

ABSTRACT

In preferred embodiments, apparatus for and method of eye tracking, including, in sequence, the steps of: viewing an entire face of a person to obtain predetermined facial features of the person to identify or not the said person; if the person is identified, retrieving a previously stored ocular profile of the person based on said predetermined facial features; using said ocular profile to track movement of an eye of said person. If the person is not identified, an ocular profile is created.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to eye tracking systems generally and,more particularly, but not by way of limitation, to novel eye trackingsystem with a larger field of view and a higher resolution imager.

2. Background Art

While the present invention is described generally in the context ofdetermining inattentiveness of the driver of a vehicle, it will beunderstood that the invention is application, as well, to any situationin which tracking of eye movements is desirable.

Communities in which eye tracking is used include the medical, research,and academic communities where eye tracking is useful, for example, indetermining psychological response and how a person scans.

A large number of the approximately 35,000 vehicular deaths and a muchlarger number of vehicular accidents each year in the United States canbe attributed to driver inattention. Driver inattention may result fromdrowsiness, use of a telephone, use of a computer, changing stations ona radio, conversing with another occupant of the vehicle, and similarcauses. Whatever the cause, such driver inattention may result infatalities and non-fatal accidents.

In response to this problem, systems have been developed to continuouslytrack an eye of a driver and use this information to determine when thedriver becomes inattentive and to institute measures to alert the driverof the potential disaster. Eye tracking can be an accurate,non-intrusive, non-contact method, which detects driver inattention andfatigue by recording eye movement. Parameters such as blink duration,blink frequency, and eye closure can be useful for detecting drowsiness,fatigue, and other causes of inattentiveness. Pupillary data such asdiameter, shape, location, and rate of change can be useful indetermining the direction of the driver's gaze and workload. Thus, it isa key safety system that can be used to determine the state of thedriver for workload management systems.

Although there are a number of eye tracking methods, many are quiteexpensive and many involve undesirable intrusiveness, both undesirablefeatures for a vehicular system. Some of these methods are discussedbelow.

The limbus eye tracking method utilizes the difference in reflectancebetween the sclera and the iris. The reflectively difference produces acontrast that can be monitored by photodetectors mounted on either sideof the eyeball. This method requires head mounted sensors and, when usedalone, is sensitive to head movement when calculating gaze.

The pupil tracking method is similar to the limbus tracking method,except that it uses the boundary between the pupil and iris, rather thanthe boundary between the sclera and the iris. This method isadvantageous in that the pupil is far less covered by the eyelid than isthe limbus and the border is sharper than that of the limbus, thusoffering a higher resolution. A disadvantage is that the contrast islower.

The cornea/pupil relationship method may be one of two types. In darkpupil tracking, collimated, on-axis IR illumination causes a dark pupilto appear bright. The high contrast with the dark pupil acts as a lightsink. In bright pupil tracking, uncollimated, off-axis IR illuminationreflects off the retina similar to the reflection seen from the eyes ofa nocturnal animal or in red-eye from flash photography. In either case,a CCD or CMOS detector is used, Pupil edge is located and the center iscalculated.

The artificial neural network method uses brain waves and artificialintelligence in software to calculate alertness, etc. Capturing thebrain waves requires the use of electrodes placed on the scalp orplacing the subject in an MRI type of device

The dual Purkinje image method compares the corneal reflection to thereflection from the back surface of the lens. Measurements are made ofthe relative displacement between the first and fourth reflections, tothe shape and size of the pupil, which represent the focal point.Because the IR illumination has to travel through seven interfaces, themethod is very inefficient and requires a relatively powerfulillumination source.

The electro-oculographic method calculates eye position by measuring thepotential difference between the front and back of the eyeball. Thismethod is simple, but not reliable, due to signal variations over time,and not precise, due to noise levels. Of course, electrodes must beplaced around the eye.

The search coil method requires that induction coils be embedded in thesclera or in tight fitting contact lenses. This is a very accuratemethod, but also quite intrusive.

The video-oculographic method gives an accurate measurement of eyecomponents, including ocular torsion. The method also provides pupildiameter measurement and monocular and binocular measurement with pupildiameter. There is no drift and reduced noise sensitivity.

As noted above, the foregoing methods are unsuitable for vehicular usebecause of cost and/or intrusiveness, although some produce fairlyaccurate results.

Commonly, a system used in vehicular eye tracking includes one or two IRsources directed at an eye of the driver, a combination of mirrors andmotors, an eye camera module, and a processing and control unit. As thetwo light sources shine on an eye, two spots (invisible, but seen by theinfrared camera) occur in the pupil. The eye camera module captures andsends an image of the eye to the processing and control unit, whichcalculates the direction of driver gaze by comparing the locations ofthe two spots in the pupil. The mirror/motor combination is used infront of the eye camera adjusts to try to track the driver's eye as thedriver's head is moved and also adjusts to driver height throughcalibration. This requires a complex and expensive auto focus andtracking system.

One problem is that such current eye tracking systems use a small fieldof view lens to cover a small area around the eye of the driver and animage sensor having a sub-QVGA (“quarter video graphic array) resolution(or one-quarter of the standard 640×480 VGA pixelation). Because ofthese limitations, conventional eye tracking systems permit only limitedhead movement. If the limited degree of head movement is exceeded bysudden head movement, head rotation, or vibration, for example if thevehicle runs on a rocky road, the system will “lose” the eye and becomesineffective until the eye can again be tracked which can take some time.

Another problem with such current eye tracking systems is that, sinceeach person has a slightly different ocular profile that can result in alarge error in calculation, the eye tracking system needs to becalibrated for each different driver. This is done by having the driversit still in the driver's seat, while an instructor sits in the frontpassenger seat. The instructor tells the driver to look at a particularspot, or fixate on predetermined alignment coordinates, for a shortperiod of time. The instructor then measures and records oculometricdata. This procedure is repeated some six or nine times. Then, theinstructor takes the recorded data and develops an ocular profile of thedriver that is recorded in a look-up table that is manually accessed foreach driver. This must be done for each potential driver of the vehicle.In addition to the fairly tedious start-up procedure, periodicrecalibration is required because of drift.

In addition to the above problems, the use of electromechanicalcomponents inherently renders a system complex and relatively costly andintroduces additional areas in which alignment is critical and in whichthe wearing of mechanical linkages eventually introduces additionalerrors into the eye tracking system. Accordingly, it is a principalobject of the present invention to provide an eye tracking system thathas no mechanical linkages.

It is a further object of the invention to provide such an eye trackingsystem that can increase the field of view to well beyond the areaaround the eye, while maintaining adequate resolution.

It is an additional object of the invention to provide such an eyetracking system that is automatically calibrated for each driver.

It is another object of the invention to provide such an eye trackingsystem that permits nearly 180° head rotation, while continuing tomonitor an eye.

It is yet a further object of the invention to provide such an eyetracking system that includes automatic driver identification.

It is yet an additional object of the invention to eliminate or greatlyreduce the potential for “lost” eye contact.

It is yet another object of the invention to provide such an eyetracking system that uses high resolution methods to examine the entireface of a driver and retrieve and ocular profile based on theidentification of the driver or to prepare and store an ocular profile,using a high resolution image of an eye, if the driver is notrecognized. Other objects of the present invention, as well asparticular features, elements, and advantages thereof, will beelucidated in, or be apparent from, the following description and theaccompanying drawing figures.

SUMMARY OF THE INVENTION

The present invention achieves the above objects, among others, byproviding, in preferred embodiments, apparatus for and method of eyetracking, comprising, in sequence, the steps of: viewing an entire faceof a person to obtain predetermined facial features of said person toidentify or not said person; if said person is identified, retrieving apreviously stored ocular profile of said person based on saidpredetermined facial features; using said ocular profile to trackmovement of an eye of said person. If the person is not identified, anocular profile is created.

BRIEF DESCRIPTION OF THE DRAWING

Understanding of the present invention and the various aspects thereofwill be facilitated by reference to the accompanying drawing figures,provided for purposes of illustration only and not intended to definethe scope of the invention, on which:

FIG. 1 is schematic top plan view of an eye tracking system according tothe present invention used to track the eye movements of a person.

FIG. 2 shows the reflections of IR sources in an eye

FIG. 3 shows a full face view and a view of an eye as used in thepresent invention.

FIG. 4 shows various features that may be used by the present inventionto identify a driver

FIG. 5 is a logic flow diagram of the system of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference should now be made to the drawing figures on which similar oridentical elements are given consistent identifying numerals throughoutthe various figures thereof, and on which parenthetical references tofigure numbers direct the reader to the view(s) on which the element(s)being described is (are) best seen, although the element(s) may be seenon other figures also.

FIG. 1 illustrates an example of a system, according to the presentinvention, and generally indicated by the reference numeral 10. System10 includes first and second IR sources 20 and 22 disposed so as toilluminate at least the area around one eye 24 in the face 26 of a user(only the face shown). System 10 also includes a sensor module 30disposed so as to receive reflections from the eye and to trackmovements of eye 24. Sensor module 30 could include electronic pan tiltto compensate for head and eye movement and the sensor module providesinput to processing and control circuitry in a processing platform 40,the processing and control circuitry also controlling the other elementsof system 10. Illumination sources 20 other than IR may be provided, aslong as the other illumination sources are non-intrusive. System 10 hasno moving parts.

The conventional, low-resolution CCD sensor in sensor module 30 has beenreplaced in this invention with a high-resolution CMOS device, whichpermits a larger field of view, while maintaining the minimum requiredpixel density for precise oculometric data acquisition and permitsincreased freedom of head movement. This change increases the resolutionof system 10 from the conventional sub-QVGA to high resolution such asSXGA (1280×1024 pixels). Now, the entire face 26 of the user can beviewed, permitting the use of known techniques to extract facialfeatures after suitable image processing. This results in the 8-10pixels per degree that is achieved with conventional systems that viewonly the eye of the driver.

The facial feature extraction enables the system to recognize the driverof whom the oculometric profile is stored and uploaded quickly. Thisthen eliminates the need for annoying recalibration for differentlearned drivers, as well as expedites the learning process. Facialfeature extraction also enables auto-calibration by the use of searchand recognition algorithms to determined eye location and its associatedoculometric geometry within system 10. Facial feature extraction canalso be used to determine how the vehicle can be operated or access tovarious areas of the vehicle. For example, if the driver is notrecognized, the operating system may go into “valet mode” in whichcertain areas of the vehicle, such as the glove compartment or the trunkof the vehicle cannot be accessed by the driver.

FIG. 2 illustrates the reflections 60 of IR sources 20 (FIG. 1) in theeye 24 of a driver. It will be understood that the reflections shown onFIG. 2 are for illustrative purposes only and that the reflections wouldnot be visible. This is an example of a non-contact technique that usesthe pupil/cornea reflection principle. The positions of the pupil andreflex centers are determined from a digitized video image and thedriver's point of gaze is determined by calculating the differencevectors of reflections 60.

FIG. 3 illustrates the full face view and an eye view as used by system10 (FIG. 10), the use of which is described below.

FIG. 4 illustrates facial features that a configuration similar tosystem 10 (FIG. 1) may use to identify a driver. These include thepositions and sizes of the eyes 70, the corners 80 of the eyes, the ears90, the corners 100 of the mouth, the chin 110, and the ala 120, of thenose.

Initial calibration now requires only that a driver sit in the driver'sseat. System 10 automatically scans the face of the driver and storesidentifying information in memory. Now, when the same driver sits in thedriver's seat, system 10 scans the face of the driver, recognizes thedriver and automatically uploads the driver's profile, eliminating theneed to re-calibrate for different drivers. This profile can also beused to identify the driver and load stored presets. Of course, portionsof system 10 may be overridden if, for example, the vehicle is to begiven to a parking valet.

FIG. 5 is a logic flow diagram illustrating the operation of system 10(FIG. 1).

First, using SXGA resolution, the whole face of a driver constitutes thefield of view The driver enters the vehicle at step 200. At step 210,the system searches for particular facial features. At step 220, thesystem acquires the facial features of the driver (FIG. 4). At step 230,the system determines if the driver is recognized. If the driver isrecognized, the driver's ocular profile is retrieved at step 240 fromprofile storage 250. This is used at step 260 to perform oculometriccalculations. At step 270, oculometrics are determined while continuingto use the whole face of the driver in the field of view.

If the driver is not recognized at step 230, the system uses SXGAresolution in which an eye of the driver fills the entire field of viewand, at step 280, the system searches for ocular features. At step 290,the system acquires the ocular features. At step 300, the systemcalibrates and creates an ocular profile. The ocular profile created atstep 300 is then categorized at step 310 with the facial featuresacquired at step 220 and the ocular profile is stored in the profilestorage 250.

At the foregoing steps are automatic and require no manual input on thepart of the driver or any other person.

Should eye contact be “lost”, by the driver completely moving thedriver's head, for example, the system goes into standby mode until aface is recognized, with the system using the whole face in the field ofview. Then, knowing where the eye being tracked is located on the face,the system can again use the ocular profile to perform oculometriccalculations. The system differentiates between a normal blink and theloss of data.

In the embodiments of the present invention described above, it will berecognized that individual elements and/or features thereof are notnecessarily limited to a particular embodiment but, where applicable,are interchangeable and can be used in any selected embodiment eventhough such may not be specifically shown.

Terms such as “upper”, “lower”, “inner”, “outer”, “inwardly”,“outwardly”, “vertical”, “horizontal”, and the like, when used herein,refer to the positions of the respective elements shown on theaccompanying drawing figures and the present invention is notnecessarily limited to such positions.

It will thus be seen that the objects set forth above, among thoseelucidated in, or made apparent from, the preceding description, areefficiently attained and, since certain changes may be made in the aboveconstruction without departing from the scope of the invention, it isintended that all matter contained in the above description or shown onthe accompanying drawing figures shall be interpreted as illustrativeonly and not in a limiting sense.

It is also to be understood that the following claims are intended tocover all of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall therebetween.

1. A method of eye tracking, comprising, in sequence, the steps of: (a)viewing an entire face of a person to obtain predetermined facialfeatures of said person to identify or not said person; (b) if saidperson is identified, retrieving a previously stored ocular profile ofsaid person based on said predetermined facial features; (c) using saidocular profile to track movement of an eye of said person.
 2. A methodof eye tracking, as defined in claim 1, wherein: said steps areperformed automatically.
 3. A method of eye tracking, as defined inclaim 1, wherein said step of using said ocular profile views saidentire face of said person.
 4. A method of eye tracking, as defined inclaim 1, wherein: if said person is not recognized, such information isused to restrict how a vehicle is to be operated and/or to restrictaccess to certain areas of a vehicle.
 5. A method of eye tracking, asdefined in claim 1, wherein: following step (a), if said person is notrecognized, viewing only an eye of said person to create an ocularprofile of said eye and using said ocular profile in step (c).
 6. Amethod of eye tracking, as defined in claim 1, wherein: steps (a) and(c) are performed using SXGA resolution in a eye tracking system.
 7. Amethod of eye tracking, as defined in claim 5, wherein said step ofcreating an ocular profile uses SXGA resolution in an eye trackingsystem.
 8. A method of eye tracking, as defined in claim 1, wherein saidstep of using said ocular profile includes illuminating said eye withinfrared radiation and using reflection of said infrared radiation foreye tracking.
 9. A method of eye tracking, as defined in claim 1,wherein: if eye contact is lost, viewing said entire face of said personto determine the location of said eye.
 10. An apparatus for eyetracking, comprising: (a) an optical receiver to view an entire face ofa person to obtain predetermined facial features and to produce a signalrepresentative of said facial features; (b) infrared illumination tocause a reflection from an eye of said person; and (c) a controller toreceive said signal and, if said person is recognized from said facialfeatures, to retrieve a stored ocular profile of said reflection and touse said ocular profile for eye tracking while viewing said entire faceof said person.
 11. An apparatus for eye tracking, as defined in claim10, wherein: if said person is not recognized, said controller causessaid optical receiver to view only said eye of said person and uses suchinformation to create an ocular profile therefor for use in eyetracking.
 12. An apparatus for eye tracking, as defined in claim 10,wherein: all elements of said apparatus operate automatically.
 13. Anapparatus for eye tracking, as defined in claim 10, wherein: allelements of said apparatus operate without mechanical linkages.
 14. Anapparatus for eye tracking, as defined in claim 10, wherein: saidapparatus uses SXGA resolution.
 15. An apparatus for eye tracking, asdefined in claim 10, wherein: if said person is not recognized, saidcontroller restricts how a vehicle can be operated and/or restrictsaccess to certain areas of said vehicle.
 16. An apparatus for eyetracking, as defined in claim 10, wherein: if contact with said eye islost, said controller uses said entire face of said person to locatesaid eye.